Skip to content

feat: configurable interview panel size with stratified sampling - #28

Open
jcartagenamartinez-legaltech wants to merge 8 commits into
amadad:mainfrom
jcartagenamartinez-legaltech:feat/stratified-interviews
Open

feat: configurable interview panel size with stratified sampling#28
jcartagenamartinez-legaltech wants to merge 8 commits into
amadad:mainfrom
jcartagenamartinez-legaltech:feat/stratified-interviews

Conversation

@jcartagenamartinez-legaltech

Copy link
Copy Markdown

Summary

The post-simulation interview panel is hardcoded to the 5 most active agents.
This caps the qualitative evidence the report can draw on and biases it toward
the vocal minority — the silent agents, who watched the whole simulation and
engaged with none of it, often hold the most informative objections ("why
didn't this interest you?" is half of any real market research).

  • MIROFISH_INTERVIEW_MAX_AGENTS (default 5, current behavior
    preserved) controls panel size. Cost scales linearly — one LLM call per
    agent per platform.
  • With activity data and a panel of ≥6, selection becomes stratified:
    ~40% most vocal, ~30% LLM-picked for topical relevance from the remainder,
    ~30% silent non-participants sampled across the roster. Below the threshold
    the existing LLM-selection path is unchanged.
  • MIROFISH_INTERVIEW_EXTRA_QUESTIONS (;-separated) appends fixed
    questions to the generated questionnaire — useful for closed questions
    (price bands, adoption timing) whose answers can be coded into
    distributions for quantitative downstream use.

Stacked on #26

Branched from #26 because a larger panel is only viable with its scaled batch
timeout (a flat 180s/600s budget starves any panel beyond a handful of
agents). Merge #26 first and this rebases clean; the interview-related diff
of this PR is the last commit only.

Testing

  • Unit smoke of the stratified selector: quotas honored (4/3/3 on a
    10-agent panel over 20 profiles), no duplicate indices, all in range,
    most-active agents always included, classic path below threshold, and the
    no-LLM fallback still deterministic.
  • py_compile clean on the three touched modules.
  • Defaults unchanged: without the env vars, behavior is identical to today.

🤖 Generated with Claude Code

Two issues prevented the claude-cli LLM provider from running on
Windows:

- cwd="/tmp" is a POSIX-only path; on Windows this raises
  WinError 267 (invalid directory name). Replaced with
  tempfile.gettempdir() for both the claude and codex subprocess
  calls, which resolves correctly on all platforms.

- The prompt was passed as a trailing argv element
  (["claude", "-p", "--output-format", "json", prompt]). Large
  prompts (tens of KB, common once a few input files are attached)
  exceed Windows' command-line length limit and raise
  WinError 206 ("filename or extension is too long"). Passing the
  prompt via stdin instead removes this ceiling. Bumped the
  subprocess timeout from 300s to 600s to give slower/CLI-based
  providers headroom, and set encoding="utf-8", errors="replace"
  explicitly since Windows' default locale encoding (cp1252) was
  raising UnicodeDecodeError on non-ASCII input/output.

Tested on Windows 11 running full simulations (100+ agents,
15 rounds) end to end with LLM_PROVIDER=claude-cli.
rapidfuzz's latest release at time of writing (3.14.4) has no
win_amd64 wheel published, which makes `uv sync` fail outright on
Windows (no sdist build toolchain assumed). Overriding to 3.13.0,
the latest version with a Windows wheel, unblocks installation.

This can be relaxed/removed once upstream publishes Windows wheels
for a newer version again.
interview_agents() (used by the report generator to interview the
top N most-active agents post-simulation) sends a batch IPC command
with a hardcoded 180s timeout covering up to N agents x 2 platforms
of individual LLM interview calls.

With CLI-based LLM providers this was consistently timing out with
0/N successful interviews on runs with 100+ agent profiles, even
though the simulation itself completed normally — the report would
silently fall back to "(no interview data)" since the failure is
caught as non-fatal by the caller.

180s doesn't leave enough headroom when each individual call can
legitimately take up to the LLM client's own per-call ceiling (600s
in llm_client.py, e.g. for claude-cli subprocess spawns). Raised to
match that ceiling.
Addresses self-review findings on the original three commits:

- Single CLI_CALL_TIMEOUT_SECONDS constant governs both providers and
  every timeout error message (the claude handler still said "300s"
  after the ceiling moved to 600; codex was left at 180 with no
  encoding handling — same cp1252 hazards the claude path already
  fixed).
- Interview batch budget now scales with the batch instead of a flat
  600s: one full per-call ceiling per interviewed agent. A fixed
  budget equals the worst case of a single call and still starves
  multi-agent batches.
- rapidfuzz override scoped with sys_platform == 'win32' marker so
  macOS/Linux keep the resolver's pick; pyproject now documents why
  the pin exists and when to drop it (the missing wheels are
  specifically Windows cp312, not all of win_amd64).
- Provider binaries resolved via shutil.which for a clear early error
  when the CLI is missing (and to tolerate npm-style .cmd shims on
  Windows).
- Warn when undecodable bytes were replaced with U+FFFD instead of
  corrupting output silently.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
The post-simulation interview panel was hardcoded to the 5 most active
agents, which (a) caps the qualitative evidence the final report can
draw on and (b) over-represents the vocal minority — silent agents,
who watched the whole simulation and engaged with none of it, often
hold the most informative objections.

- MIROFISH_INTERVIEW_MAX_AGENTS env var (default 5, preserving current
  behavior) controls panel size. Cost scales linearly: one LLM call per
  agent per platform; the batch timeout already scales with panel size.
- With activity data and a panel of >=6, selection becomes stratified:
  ~40% most vocal, ~30% LLM-picked for topical relevance from the
  remainder, ~30% silent non-participants sampled across the roster.
  Below the threshold the existing LLM selection path is unchanged.
- MIROFISH_INTERVIEW_EXTRA_QUESTIONS (';'-separated) appends fixed
  questions to the generated questionnaire — closed questions (price
  bands, adoption timing) whose answers can be coded into
  distributions for downstream quantitative use.

Tested: stratified selector unit-smoke (quotas, no duplicates, range,
classic path below threshold, no-LLM fallback) plus py_compile.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
- LLMTimeoutError (RuntimeError subclass) raised on TimeoutExpired and
  excluded from the x3 retry loop: a prompt that exhausts the 600s
  ceiling will almost certainly exhaust it again, so retrying turned
  one slow call into ~30 minutes of blocked wall clock.
- _resolve_cli cached with lru_cache: the binary does not move during
  the process lifetime, so the per-request PATH scan was waste.
  (Failures are not cached: lru_cache does not memoize exceptions.)

🤖 Generated with [Claude Code](https://claude.com/claude-code)
…dedup

- _get_int_env helper: malformed MIROFISH_INTERVIEW_MAX_AGENTS values
  degrade to the default with a warning instead of crashing at import
  from the Config class body; floor of 1 guards zero/negative panels.
- Interview batches now run in chunks of 5 with a per-chunk timeout
  (ceiling x chunk size): the previous single all-or-nothing IPC
  transaction scaled to hours with large panels and lost every
  completed interview if one hung. A failed chunk now loses only
  itself; results accumulate across chunks.
- LLM-selected interview indices are deduped (order-preserving),
  coerced from strings and range-checked before use.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant